Three-dimensional (3D) cameras based on time-of-flight (TOF) principle acquire distance information from object(s) in a scene being imaged. Distance information is produced independently at each pixel of the camera's sensor array. Exemplary such systems are described in U.S. Pat. No. 6,323,942 “CMOS-Compatible Three-Dimensional Image Sensor IC” (2001), and U.S. Pat. No. 6,515,740 “Methods for CMOS-Compatible Three-Dimensional Image Sensing Using Quantum Efficiency Modulation” 2003, which patents were originally assigned to Canesta, Inc., and are now assigned to Microsoft, Inc.
As described in U.S. Pat. No. 6,323,942, a TOF system emits optical energy (active optical energy) and determines how long it takes until at least some of that energy, reflected by a target object, arrives back at the system to be detected by the array of sensors. Emitted optical energy traversing to more distant surface regions of a target object before being reflected back toward the TOF system will have a longer TOF than if the target object were closer to the system. If the roundtrip TOF time is denoted t1, then the distance between target object and the TOF system is Z1, where Z1=t1·C/2, where C is velocity of light. Such systems can acquire both luminosity data (signal amplitude) and TOF distance, and can realize three-dimensional images of a target object in real time. Advantageously such systems can operate with or without ambient light (passive optical energy), and can operate without moving parts, for example, shutters, motors.
A more sophisticated TOF system is described in U.S. Pat. No. 6,515,740, wherein TOF is determined by examining relative phase shift between modulated transmitted light signals and light signals reflected from a target object. FIG. 1A depicts an exemplary phase-shift detection system 100 according to the '740 patent. Detection of the reflected light signals over multiple locations in the system pixel array results in measurement signals that are referred to as depth images. The depth images represent a three-dimensional image of the target object surface.
Referring to FIG. 1A, TOF system 100 includes a two-dimensional array 130 of pixel detectors 140, each of which has dedicated circuitry 150 for processing detection charge output by the associated detector. In a typical application, array 130 might include 100×100 pixels 230, and thus include 100×100 processing circuits 150. IC 110 may also include a microprocessor or microcontroller unit 160, memory 170 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 180, and various computing and input/output (I/O) circuitry 190. Among other functions, controller unit 160 may perform distance to object and object velocity calculations.
Under control of microprocessor 160, a source of optical energy 120 is periodically energized via exciter 115, and emits optical energy via lens 125 toward an object target 20. Typically the optical energy is light, for example emitted by a laser diode, VCSEL (vertical-cavity surface emitting laser) or LED device 120. Some of the optical energy emitted from device 120 will be reflected off the surface of target object 20, and will pass through an aperture field stop and lens, collectively 135, and will fall upon two-dimensional array 130 of pixel detectors 140 where an image is formed. In some implementations, each imaging pixel detector 140 captures time-of-flight (TOF) required for optical energy transmitted by emitter 120 to reach target object 20 and be reflected back for detection by two-dimensional sensor array 130. Using this TOF information, distances Z can be determined. Advantageously system 100 can be implemented on a single IC 110, with relatively few off-chip components.
Typically optical energy source 20 emits preferably low power (e.g., perhaps 1 W peak) periodic waveforms, producing optical energy emissions of known frequency (perhaps 30 MHz to a many hundred MHz) for a time period known as the shutter time (perhaps 10 ms). Optical energy from emitter 120 and detected optical energy signals within pixel detectors 140 are synchronous to each other such that phase difference and thus distance Z can be measured for each pixel detector. The detection method used is referred to as homodyne detection in the '740 and '496 patents. Phase-based homodyne detection TOF systems are also described in U.S. Pat. No. 6,906,793, Methods and Devices for Charge Management for Three-Dimensional Sensing, originally assigned to Canesta, Inc., and now assigned to Microsoft, Inc., assignee herein. Applicants incorporate said '793 patent herein by reference.
The optical energy detected by the two-dimensional imaging sensor array 130 will include light source amplitude or intensity information, denoted as “A”, as well as phase shift information, denoted as φ. As depicted in exemplary waveforms in FIGS. 1B and 1C, the received phase shift information (FIG. 1C) varies with TOF and can be processed to yield DATA, including Z data. For each pulse train of optical energy transmitted by emitter 120, a three-dimensional image of the visible portion of target object 20 is acquired, from which intensity and Z data is obtained (DATA). As described in U.S. Pat. Nos. 6,515,740 and 6,580,496 obtaining depth information Z requires acquiring at least two samples of the target object (or scene) 20 with 90° phase shift between emitted optical energy and the pixel detected signals. While two samples is a minimum figure, preferably four samples, 90° apart in phase, are acquired to permit detection error reduction due to mismatches in pixel detector performance, mismatches in associated electronic implementations, and other errors. On a per pixel detector basis, the measured four sample data are combined to produce actual Z depth information data. Further details as to implementation of various embodiments of phase shift systems may be found in U.S. Pat. Nos. 6,515,740 and 6,580,496.
FIG. 1D is similar to what is described with respect to the fixed phase delay embodiment of FIG. 10 in U.S. Pat. No. 6,580,496, entitled Systems for CMOS-Compatible Three-Dimensional Image Sensing Using Quantum Efficiency Modulation, or in U.S. Pat. No. 7,906,793, entitled Methods and Devices for Charge Management for Three-Dimensional Sensing, both patents originally assigned to Canesta, Inc., and now assigned to Microsoft, Inc., assignee herein. (Applicants incorporate the '496 and the '793 patents herein by reference.) In FIG. 1D, generated photocurrent from each quantum efficiency modulated differential pixel detector, e.g., 140-1, is differentially detected (DIF. DETECT) and differentially amplified (AMP) to yield signals B·cos(φ), B·sin(φ), where B is a brightness coefficient.
During normal run-time operation of the TOF system, a fixed 0° or 90° phase shift delay (DELAY) is switchably insertable responsive to a phase select control signal (PHASE SELECT). Homodyne mixing occurs using quantum efficiency modulation to derive phase difference between transmitted and received signals (see FIGS. 1B, 1C), and to derive TOF, among other data. A more detailed description of homodyne detection in phase-based TOF systems is found in the '496 patent. Although sinusoidal type periodic waveforms are indicated in FIG. 1D, non-sinusoidal waveforms may instead be used. Detection configurations similar to FIG. 1D may be used with embodiments of the present invention.
Thus, TOF systems as exemplified by FIG. 1A can acquire phase delay (φ) between the modulated transmitted light and target object-reflected light. While phase φ is proportional to the (Z) distance separating the TOF sensor array and the target object, phase delay is a relative quantity and is not per se equal to Z distance. For example as Z increases, phase φ increases, but after an increase of 360°, the phase folds-over and further increases in Z will produce further increases in φ, again starting from 0°. In practice, one generally provides TOF systems with the ability to disambiguate or de-alias the phase data to obtain a true measure of Z.
As exemplified by U.S. Pat. No. 7,719,662 originally assigned to Canesta, Inc., and now assigned to Microsoft, Inc., assignee herein, it is beneficial to further provide a TOF system with a calibration method enabling use of Cartesian (or real world X, Y, Z) coordinates instead of radial information. (TOF pixels or sensors measure phase delay along a certain radial angle that is different for each pixel 140 in array 130.) According to the '662 patent, one function of calibration may be defined as creating a mapping from sensor 140 response to geometrical coordinates, which are X, Y, and Z information with respect to a known reference. (X and Y coordinates are the horizontal and vertical offsets from the optical axis of the TOF system, and Z is the perpendicular distance between the sensor and the target object or object within an imaged scene.) Such mapping may include depth calibration as well as XY calibration. The '662 patent disclosed calibration mapping methods that did not require gathering depth data at different values of Z, as a target object was physically relocated to the different Z depths in the XY plane (the so-called “by example” method). Data so gathered could be collected and stored in a look-up-table (LUT) that was available to the TOF system during run time operation to make corrections in acquired depth data.
In addition to geometrical calibration, one must perform other types of calibration to account for certain environmental factors, including without limitation temperature and ambient lighting conditions. For example, temperature changes in sensor array 130 can increase so-called dark current in pixels 140, which dark current can in turn change measured phase φ. Ambient light can interfere with system-emitted light from source 120, and can result in phase errors. A complete calibration procedure preferably will include steps to model the effects of such environmental changes. So doing can allow these effects to be removed dynamically during run-time operation, when the environmental conditions may change.
Thus, a given phase response from the sensor array is converted to distance by interpolating the values stored in the calibration table. However the phase-vs-distance transfer function curve contains harmonics and sufficient data points must be stored in the calibration table to model these harmonics to avoid loss of accuracy due to insufficient sampling. There is also interpolation error that can only be reduced by increasing the size of the table.
Although the “by-example” method is straightforward to implement with relatively fast run-time processing, it has several disadvantages. Taking a subset of the operating range and subsequent interpolation results in errors that can be several cm in magnitude. Further, as the operating range of the sensor is increased, more data must be stored in the calibration table to maintain accuracy. This generates larger calibration tables, requiring more storage, as well as longer interpolation times. Storage can be on the order of several MB, e.g., very large for use with embedded systems. Another problem from a practical standpoint is the large physical space needed to capture data from the sensor for large field of view (FOV) and operating ranges as the target object is repositioned. For example, a sensor with a 100° FOV and 5 m operating range requires a target object of approximately 12 m×12 m, which target object must be moved between 0 and 5 m during calibration. Given enough physical space for target object relocation during calibration, and given enough time for the calibration procedure, such prior art “by example” calibration can be carried out. But such prior art calibration procedure has high costs and is not very suitable for calibrating a high-volume product.
Statically storing calibration correction terms in a lookup table (LUT) has been a traditional approach to calibration. However LUT coefficient values that were created when a TOF system was new, may no longer provide good calibration correction as the TOF system ages, or individual components change, e.g., with temperature. Further storing LUT values can increase system memory storage overhead.
U.S. Pat. No. 7,719,662 addressed several efficient methods and systems to implement detected phase to distance calibration for three-dimensional camera systems. The '662 patent described methods and systems requiring less time and smaller physical space to be carried out. However further improvement is generally desirable.
There are tradeoffs in any TOF system design. On one hand, emitting optical energy waveforms that are square-waves advantageously enhances modulation contrast but the fast rise and fall times that characterize square-waves contribute to undesired high order harmonics.
What is needed are methods and systems to reduce error due to harmonic content in the phase-based data acquired by a TOF system, sometimes referred to as bias error. Preferably such methods and systems would advantageously enable high modulation contrast associated with emitted square-wave optical energy waveforms, while compensating for the bias error attendant with the higher order harmonics that accompany such waveforms. Changes in bias error can result from variations in the waveform of the emitted optical energy due to time and temperature. It then becomes necessary or at least desirable to compensate for change in bias error by changing the calibration requirements for the TOF system. Preferably such methods and systems should be operable dynamically without increasing, and preferably by decreasing, memory requirements, while preserving high modulation contrast. Implementation of such methods and systems would provide dynamically good calibration of TOF systems with enhanced linearity even over varying system and/or environmental conditions.
The present invention provides such methods and systems.